Holographic Technology Implemented Retail Solutions

ABSTRACT

Disclosed are techniques that use mixed reality, e.g., augmented reality and virtual reality technologies to improve analysis of retail processes and activity in retail establishments. A server system receives images from a mixed reality device, which images including representations of physical objects and receives voice data from the mixed reality device and executes a cognitive agent to process the voice data received from the mixed reality device that generates a set of virtual objects that include data describing merchandising process concepts regarding the set of physical objects, based on input queries received in the voice data and sending by a server system, the set of virtual objects to the mixed reality device.

CLAIM OF PRIORITY

This application claims priority under 35 U.S.C. §119(e) to provisionalU.S. Patent Application 62/361,053, filed on Jul. 12, 2016, entitled:“Holographic Technology Implemented Security and Retail Solutions” theentire contents of which is incorporated herein by reference andprovisional U.S. Patent Application 62/361,669, filed on Jul. 13, 2016,entitled: “Holographic Technology Implemented Security and RetailSolutions the entire contents of which is incorporated herein byreference.

HOLOGRAPHIC TECHNOLOGY IMPLEMENTED RETAIL SOLUTIONS Background

This description relates to operation of sensor networks such as thoseused with security, intrusion and alarm systems installed on commercialpremises.

It is common for businesses to have systems for detecting conditions attheir premises and signaling the conditions to a monitoring station orto authorized users of the security system. For example, such buildingsemploy systems in the areas of fire detection, smoke detection,intrusion detection, access control, video surveillance etc. Manydifferent types of security sensors are deployed in commercialbuildings. Types of sensors typically include motion detectors, cameras,and proximity sensors (used to determine whether a door or window hasbeen opened). Such sensors can constantly collecting data that is usedto determine whether an alarm should be triggered, but also continues tocollect data after an alarm is triggered.

Retail establishments often use simple physical walk-throughs with usershaving smart-phone and/or tablet based presentations, and useconventional retail analytics applications, and verbal descriptions astools used for analysis to investigate trends and potential explanationsof observations suggested by data analytics.

Augmented reality, virtual reality and mixed reality technologies areknown. Generally, virtual reality refers to technologies that replicatean environment with a simulation of a user being immersed in thereplicated environment. Augmented reality, generally refers totechnologies that present a view of a real-world environment augmentedwith computer generated data. Mixed reality a relatively new termgenerally involves technologies that involve a merging of real world andvirtual world environments where real and virtual objects exist andinteract.

SUMMARY

According to an aspect, a system includes a server system including oneor more processor devices, memory in communication with the one or moreprocessor devices, and a storage device that stores a program ofcomputing instructions for execution by the processor using the memory.The program includes instructions configured to cause the processor toreceive images from a mixed reality device, which images includingrepresentations of physical objects, receive voice data from the mixedreality device, execute a cognitive agent to process the voice datareceived from the mixed reality device, determine location informationto determine a correspondence between the representations of thephysical objects in the images and stored information concerning the setof physical objects, generate a set of virtual objects that include datadescribing merchandising process concepts regarding the set of physicalobjects, based on input queries received in the voice data and send theset of virtual objects to the mixed reality device.

Aspects also include computer program products and methods.

Disclosed are techniques that use mixed reality and/or augmented realityand virtual reality technologies to improve the analysis of retailprocesses and activity in retail establishments. The disclosedtechniques use computer implemented techniques that obtain informationfrom various electronic systems/devices in the physical world, whichdevices are exemplified by security systems, and merge that informationinto a virtual world of policies and analytics that involve retailsystems that generate analytical information regarding customers andtheir preferences and needs. This improves upon simple physicalwalk-throughs blended with smart-phone and tablet based presentations,conventional retail analytics apps, and verbal descriptions. In manycases the main tools of such analysis are limited to emails andspreadsheets. Using these conventional methods it is very time consumingand difficult, or even impossible, to investigate trends and potentialexplanations of observations suggested by data analytics.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention is apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of an exemplary networked security system.

FIG. 2 is a block diagram of constrained device.

FIG. 3 is a block diagram depicting a sales promotion system integratedwith a mixed reality system.

FIG. 4 is a flow chart of an embodiment of a sales promotionapplication.

FIG. 5 is a block diagram of an AR/VR session manager.

FIG. 6 is a block diagram of an AI based cognitive agent.

DETAILED DESCRIPTION

As shown in FIG. 1, described herein are examples of an integratedplatform 10 that integrates via a distributed network 11, mixed realitydevices 13 a-13 c with security/intrusion/alarm/surveillance systems 15a-15 c (typically including sensors 20, functional nodes 18 andtypically including a panel not shown).

Examples of mixed reality devices 13 a-13 c are those in which the mixedreality devices incorporate a live, real world presentation of elementsof the physical real-world with virtual elements that are calculated orproduced from inputs and which are rendered on a display so that to auser these calculated or produced elements are perceived to existtogether with the physical real world in a common environment. Examplesof such mixed reality devices 13 a-13 c include mixed reality devicessuch as Hololens® (Microsoft), (a smart-glasses, cordless, Windows 10®(Microsoft) computer headset that includes various sensors and ahigh-definition stereoscopic 3D optical head-mounted display, andspatial sound to allow for augmented reality applications. Other mixedreality devices/augmented reality systems such as Google Glass® (Google)could be used. There are many such systems on the market of which theseare two examples.

The security systems 15 a-15 c typically include a panel (not shown),such as for an intrusion detection system, an intrusion detection panelwired or wirelessly connected to a variety of sensors deployed in apremises. Typically, such panels receive signals from one or more ofthese sensors to indicate a current state or value or that a particularcondition being monitored has changed or become unsecure.

The integrated platform 10 includes data collection systems that arecoupled to wireless sensor networks and wireless devices, with remoteserver-based monitoring via servers 14 and report generation. Asdescribed in more detail below, wireless sensor networks generally use acombination of wired and wireless links between computing devices, withwireless links usually used for the lowest level connections (e.g.,end-node device to hub/gateway 16). In an example network, the edge(wirelessly-connected) tier of the network is comprised ofresource-constrained devices 20 with specific functions. These devices20 may have a small-to-moderate amount of processing power and memory,and may be battery powered, thus requiring that they conserve energy byspending much of their time in sleep mode. A typical model is one wherethe edge devices 20 generally form a single wireless network in whicheach end-node communicates directly with its parent node (e.g., 18) in ahub-and-spoke-style architecture. The parent node may be, e.g., anaccess point on a gateway or a sub-coordinator which is, in turn,connected to the access point or another sub-coordinator.

In FIG. 1, the distributed network 11 is logically divided into a set oftiers or hierarchical levels 12 a-12 c. The mixed reality devices 13a-13 n are shown in communication with the top one or two tiers orhierarchical levels 12 a-12 c. In FIG. 1, the lower level tier 12 c isillustrated divided into different premises 19 a-19 c for ease inexplaining details of the applications that will be discussed below. Thepremises 19 a-19 c are each associated with one of the security systems15 a-15 c. The security systems can be independent meaning that thereare no connections (as shown) among fully functional nodes of differentpremises or dependent meaning that there are connections (not shown)among fully functional nodes of different premises.

In the upper tier or hierarchical level 12 a of the network are disposedservers and/or virtual servers 14 running a “cloud computing” paradigmthat are networked together using well-established networking technologysuch as Internet protocols or which can be private networks that usenone or part of the Internet. Applications that run on those servers 14communicate using various protocols such as for Web Internet networksXML/SOAP, RESTful web service, and other application layer technologiessuch as HTTP and ATOM. The distributed network 11 has direct linksbetween devices (nodes) as shown and discussed below. Servers 14 executeanalytics (analysis programs of various sorts) that are managed inconcert with a session manager system 80 (FIG. 4). The servers 14 canaccess a database 23.

The second logically divided tier or hierarchical level 12 b, referredto here as a middle tier, involves gateways 16 located at central,convenient places inside individual buildings and structures, e.g., 13a-13 c. These gateways 16 communicate with servers 14 in the upper tierwhether the servers are stand-alone dedicated servers and/or cloud basedservers running cloud applications using web programming techniques. Themiddle tier gateways 16 are also shown with both local area network 17 a(e.g., Ethernet or 802.11) and cellular network interfaces 17 b. Eachgateway is equipped with an access point (fully functional node or “F”node) that is physically attached to that access point and that providesa wireless connection point to other nodes in the wireless network. Thelinks (illustrated by lines not numbered) shown in FIG. 1 representdirect (single-hop MAC layer) connections between devices. A formalnetworking layer (that functions in each of the three tiers shown inFIG. 1) uses a series of these direct links together with routingdevices to send messages (fragmented or non-fragmented) from one deviceto another over the network.

The distributed network topology also includes a lower tier (edge layer)12 c set of devices that involve fully-functional sensor nodes 18 (e.g.,sensor nodes that include wireless devices, e.g., transceivers or atleast transmitters, which in FIG. 1 are marked in with an “F”) as wellas constrained wireless sensor nodes or sensor end-nodes 20 (marked inthe FIG. 1 with “C”). In some embodiments wired sensors (not shown) canbe included in aspects of the distributed network 11.

The distributed network 11 implements a state machine approach to anapplication layer that runs on the lower tier devices 18 and 20. Statesin the state machine are comprised of sets of functions that execute incoordination, and these functions can be individually deleted orsubstituted or added to in order to alter the states in the statemachine of a particular lower tier device. The state function basedapplication layer uses an edge device operating system that allows forloading and execution of individual functions (after the booting of thedevice) without rebooting the device (so-called “dynamic programming”).In other implementations, edge devices could use other operating systemsprovided such systems allow for loading and execution of individualfunctions (after the booting of the device) preferably without rebootingof the edge devices.

Referring to FIG. 2, a generic constrained computing device 20 that ispart of the security/intrusion/alarm/surveillance systems (eitherintegrated examples of such system or standalone examples) is shown. Aconstrained device 20 as used herein is a device having substantiallyless persistent and volatile memory other computing devices, sensors,systems in a particular networked detection/sensor/alarm system.Constrained device 20 includes a processor device 21 a, e.g., a CPU andor other type of controller device that executes under an operatingsystem, generally with 8-bit or 16-bit logic rather than the 32- and64-bit logic used by high-end computers and microprocessors. Theconstrained device 20 has a relatively small flash/persistent retailestablishment 21 b and volatile memory 21 c in comparison with other thecomputing devices on the network. Generally the persistent retailestablishment 21 b is about a megabyte of storage or less and volatilememory 21 c is about several kilobytes of RAM memory or less.

The constrained device 20 has a network interface card 21 d thatinterfaces the constrained device 20 to the network 11. Typically awireless interface card is used, but in some instances a wired interfacecould be used. Alternatively, a transceiver chip driven by a wirelessnetwork protocol stack (e.g., 802.15.4/6LoWPAN) can be used as the(wireless) network interface. These components are coupled together viaa bus structure. The constrained device 20 also includes a sensor 22 anda sensor interface 22 a that interfaces to the processor 21 a. Sensor 22can be any type of sensor type device. Typical types of sensors includetemperature, simple motion, 1- 2- or 3-axis acceleration force,humidity, pressure, selective chemical, sound/piezo-electrictransduction, and/or numerous others, implemented singly or incombination to detect complex events.

The disclosed implementations of a constrained device 20 can follow thecurrent constraints on flash/persistent storage memory and RAM memoryand less than 10-20 kilobytes of RAM/volatile memory, but can have moredepending on configuration and in some instances the operating system.These constrained devices 20 are configured in this manner; generallydue to cost/physical configuration considerations. These types ofconstrained devices 20 generally have a static software image (i.e., thelogic programmed into the constrained device is always the same).

Constrained devices 20 execute a real-time operating system that can usedynamic programming and support. The real-time operating system (“RTOS”)executes and otherwise manages a dynamic set of user-defined independentexecutable functions or tasks that are either built into a loaded image(software and RTOS that executes on the constrained device) or that aredownloaded during normal operation of the constrained device 20 or acombination of the two, with the former (built into the image) using assubroutines instances of the latter (downloaded during operation).Certain of the applications set forth below will cause systems to accessthese constrained devices 20 to upload data and otherwise control thedevices 20 according to needs of the applications.

In the examples below, a facility can be any type but is typically,e.g., a commercial, industrial, facility, with interior areas,(buildings) and exterior areas that are subject to surveillance andother types of monitoring. The buildings can be of any configuration,wide open spaces such as a warehouse, to compartmentalized facilitiessuch as labs/offices.

The retail establishment includes the plural sensors 22 (FIG. 1). In oneimplementation, a portion of the sensors 22 are r.f., hot spots or thelike through which Wi-Fi or other Internet access services are provided.Sensors 22 that are hot spots or the like capture information, as a usermoves about the retail establishment from the user's possession of themixed reality device 13 a, as will be discussed in further detail below.

Described now are techniques that allow the user of the mixed realitydevice 13 a (an augmented reality/virtual reality (AR/VR) device 13 a)to interact with the physical environment of the retail establishment,such as the retail items on the retail establishment's shelves, aislesbetween shelves, and spaces in and around the retail establishment,together with “virtual” items such as data objects that describemerchandising, promotions, inventory shrinkage, and other retail processconcepts, in a unified and simplified way.

A user of the mixed reality device 13 a may walk through a retailestablishment, examine physical items on retail establishment shelves,and at the same time (via the processing 40 (discussed below) thatintegrates retail-based analytical processing with mixed reality systemtechnology) will observe visual representations of results of executionof the retail-based analytically processing. These result can beubiquitous, meaning many or an abundant number of such executionresults.

Examples of such results can be so called “shrinkage levels” for theitem or category of items over a selected period of time, “foottraffic,” “dwell time,” “conversion,” and other retail-related data inspecific areas of the retail establishment (e.g., the aisle passing by aparticular retail item display) as a function of sales promotions of theitem. Other examples include the visual representation of thecorrelation of sales between the physical item in view and other itemsin the retail establishment or available online. Still other examplesinclude a correlation of profit of the particular item to profit ofother items, etc.

The mixed reality device 13 a facilitates coordination of communicationbetween two or more individuals discussing (in close proximity to eachother in the retail establishment, or via remote communications) retailestablishment processes, specific retail items, retail establishmentlayout issues, and so forth.

Some implementations include a cognitive agent (artificial intelligencebased assistant or “information retrieval and analytics” assistant) thatwhen used in conjunction with the mixed reality device 13 a can producea more powerful analysis tool. For example, the user may look at an itemon the retail establishment shelf while the AR/VR platform displaysvirtual objects (like pie charts, graphs, tables, etc.) giving sales,shrinkage, merchandising, and other retail information related to thatitem, and at the same time the user may (using natural spoken language)query the potentially large collection of backend information systems byasking the cognitive agent simple questions related to the real andvirtual objects on display. The cognitive agent using a web service (orother forms of database access methods, its own internal structures likerepresentations of episodic memory, domain knowledge bases, lexicons,and also external service accessible) includes analysis engines toanswer questions from the user. The combination of mixed reality device13 a and AI agent gives the user a very powerful analysis toolstimulated by an initial visual input of objects in the physical world(i.e., natural inspection of items in view and conversations with othersand/or questions to the platform).

Referring to FIG. 3, a sales promotion system 40 as shown includesplural databases. The system 40 is configured to execute a salespromotion application 70 (further described in FIG. 4). A firstdatabase, store inventory database 42, in system 40 is shown ascontaining data on retail items, including item name, SKU number, retailprice, wholesale price, location in the retail establishment (aisle no.,shelf no., slot no., planogram reference no., etc.), number of items instock, number of items on order, expected arrival date for orderedstock, inventory turnover for the item, and any other data associatedwith a given retail item. Store inventory database 42 is connected tothe Internet 63 (or a private network), via store inventory web services42 a.

The system 40 also includes other databases that include retailestablishment layout information (store layout database 44) includingretail planograms, fixture locations, layout codes and/or layout versionnames for each retail establishment address, historical and futureplanned changes in layout, etc. (connected to the Internet 63 via storelayout web service 46 a) The store layout database 44 could also includethe layout of the same aisle of location for the same retailer's retailestablishments that have the same configuration and demographics withthe highest performance, as measured in different ways.

The system 40 also includes an item information database 46 (connectedto the Internet via item information web service 46 a) and having photoimages or icon representations of retail items, retail establishmentshelf layouts, and other retail related objects. Retail establishmentperformance data, personnel information, and other retail operations andmerchandise data can be stored in a merchandizing and promotionsdatabase 48 connected to the Internet 63 via merchandizing andpromotions web service 48 a.

In addition, the system 40 includes a mobile AR/VR (augmentedreality/virtual reality) device, e.g., mixed reality device 13 a, anAR/VR session management system 80, and a wireless (e.g., Wi-Fi) network62 with wireless access points such as that shown above in FIG. 1,within the retail establishment 60. Other implementations of suchnetworks could be used.

The organization of the databases in FIG. 3 are given as examples andare somewhat simplified relative to the design and implementation ofactual enterprise-scale retail databases encountered in the commercialworld. That is, no attempt is made in the figure to show how thedatabases are fragmented and deployed for data redundancy, scalability,fast data access, and so forth. Also, the segregation of various typesof data into separate databases is simplified in FIG. 3 and it should berecognized that other database architectures can be imagined which arecompatible with and included as additional embodiments.

The mixed reality device 13 a, e.g., an “AR/VR device” (augmentedreality′ virtual reality) allows the user to see the real environmentwith data or “artificial images” imposed on a view of the realenvironment. Microsoft HoloLens and Google Glass are examples ofcommercial devices which allow this mixing of “real” and “virtual”realities as referred to herein also as mixed reality systems. The mixedreality device interacts with an outside network and the web (e.g.,using a Wi-Fi connection) and also allows for input from the user (e.g.,using hand gestures and/or voice commands).

FIG. 3 shows the various databases and the AR/VR session managementsystem 80 as remote applications (i.e., implemented in one or moreservers outside of the retail establishment). In one embodiment each ofthese is accessible via web services (such as RESTful micro-webservices) well known to those skilled in the art of distributeddatabases and mobile services.

In other embodiments, some or all of the data could be located onservers in the retail establishment. FIG. 3 does not suggest anyownership or management policy of the databases or the AR/VR sessionmanagement system, and the description specifically includes embodimentswhere functionality of the system of FIG. 3 is divided in arbitrary waysso as to allow ownership and/or management by various parties which mayor may not include the retailer as one of those parties.

Referring to FIG. 4, sales promotion application 70 integratesretail-based analytical processing with mixed reality system technologyis shown. Described below is a specific implementation of thisprocessing 70, others may be implemented. As a user of the mixed realitydevice 13 a walks through the retail establishment, the location of theuser and associated mixed reality device 13 a inside the retailestablishment is determined and tracked 72 as the user moves around theretail establishment with the mixed reality device 13 a.

Tracking 72 may be accomplished through a number of techniques includingwireless triangulation of the device, various “internal GPS”technologies (BLE, RFID, NFC, etc.) or dead-reckoning basedaccelerometer data integration. For the purposes of discussion it isonly necessary to note that the physical location of either the mixedreality device 13 a (or some other device on the person of the user,e.g., a smartphone) may be estimated to within a few feet of the user'sactual location in the retail establishment using technologies wellknown to those skilled in the art. Depending on the technology used totrack the location of the mixed reality device 13 a (or the user), othertechnology components such as cameras, beacons, and other access pointsmay be used. These components have been omitted from FIG. 3 and are notspecifically referred to in FIG. 4, for simplicity.

In the case where the actual device being tracked is not the mixedreality device, but rather some other device (such as a smart phone inthe pocket of the user), the tracked device makes its location (and byinference the location of the user and the mixed reality device 13 a)known by sending location data over the in-retail establishment wirelessnetwork to the AR/VR session manager 80. It should also be noted thatthe location of the user and mixed reality device 13 a may be determinedwithout any location determination functionality on the mixed realitydevice 13 a, and without any second device (i.e., smart phone) if someother outside system (e.g., a video surveillance system with imageanalytics capabilities able to determine location) is available and isused to track the user's location during the AR/VR session.

The user may also specify where in the retail establishment they are bysome other technique such as selecting a location on a map of the retailestablishment. In another embodiment the AR/VR system may determine itsown location by capturing the image or images of items in itssurroundings which have been previously mapped by some to the currentlocation. Using such a location to image map the mixed reality devicecan determine its own location. The “image” in such a case might be anactual image recorded in some convenient file format, or it might be anindex or set of indices derived from the image in a manner which makesthem unique to that image (i.e., an image index or hash).

During a session, the user views items and other points of interest inthe retail establishment through the mixed reality device 13 a. Based ona selected mode of operation into which the session has been placed, thelocation of the user, and determined orientation of the mixed realitydevice (i.e., what the device is facing and what items in the physicalenvironment the user is viewing through the device), the AR/VR sessionmanager 80 chooses 74 virtual items and context-relevant information toshow to the user on the display of the mixed reality device 13 a.

The AR/VR session manager 80 sends 76 the chosen virtual items andcontext-relevant information to the mixed reality device 13 a. The usermay view several items in the field of view of the mixed reality devicedisplay. The mixed reality device 13 a provides a user interface (notshown) that displays menu options that allow the user to highlight aspecific item, or a group of items, and display information for avariable period of time which is also selected using the interface menuitems. This information is sent 78 to the AR/VR session manager 80. TheAR/VR session manager 80 analyzes 80 the user highlight information todrill down to find corresponding content on the specific itemshighlighted in the display, which is sent to the mixed reality device 13a.

The user interface (not shown) can be used to enter 82 notes as the userreviews the real and virtual objects and information presented in thedisplay of the mixed reality device 13 a. While engaged in such asession as, the user may also use standard voice communications orvoice-to-chat technology available on the mixed reality device tocommunicate 84 with a second (remote) user or group of users or composeemails or text messages, etc. These actions may be part of a retailestablishment review process with extensive pre-planning or may beimpromptu as the user goes through the retail establishment in pursuitof day-to-day managerial responsibilities.

Referring now to FIG. 5, an AR/VR session manager 80 is shown. Thesession manager 80 interacts with the mixed reality device 13 a over theInternet using a “session portal” 82, e.g., a web service (applicationprogramming interface (API) or in another embodiment, a dedicated socketwith SMTP or other transfer protocol. The session portal 82 isbi-directional meaning that each of the mixed reality devices (MRS) 13a-13 c can send data to the session manager 80 and receive data from thesession manager 80. The mixed reality devices (MRS) 13 a-13 c sendupdates on their states to the session manager 80. The states of themixed reality devices 13 a-13 c are represented virtually or “mirrored”in a device state representation 84 inside the session manager 80.

Input from the mixed reality devices (MRS) 13 a-13 c to the sessionmanager 80 is used in analytic programs executed on the servers. Forexample, the camera on the mixed reality device 13 a may send an imagecontaining an area showing a retail item with its characteristicconsumer brand packaging (by which it is easily recognized byconsumers). This part of the image is identified by an input analyzer86, which relies on image libraries accessible via the web service ofthe item information database and potentially other databases exposed bythe consumer product manufacture, or other web browsers' image analyticsservices. The input analyzer 86 informs analytical manager 88 withinputs to analytic programs (not shown) executing on the servers 14. Theanalytics manager 88 uses a current mode and inputs presented to it, inorder to decide what to present (virtually) to the user on the deviceviewer and what to request of the analytics executing on the server.Information presented is produced by the analytics manager using datareceived from the various analytical programs that execute variousanalytics both conventional as well as to be developed. The session modemanager 90 monitors the mode selected by the user (as mirrored in thedevice state representation) and informs the analytics manager of theselection. Information presented is produced by the virtual contentmanager using data from the various databases accessible via webservices attached to the various external retail databases shown, by wayof example, in FIG. 8.

In FIG. 5, the session is logged by the input analyzer, including anynotes or annotations provided by the user of the mixed reality device(spoken, typed, or sent via some other mode of communication) intosession log/notes records 94 that are stored in a database as records.This locale log/record in the session manager 80 may be backed up in anexternal database (not shown) for long-term storage, reporting, andfurther analysis. This local session and long-term storage may alsoinclude a full record or “recording” of part or all of the session,rather than just the user notes.

The user may also view comparative information for this item relative toother items in the retail establishment, in its inventory category,department, or relative to all items in the retail establishment, orrelative to this item or group of items in other retail establishmentsor groups of retail establishments (e.g., the retail establishment'sregional division).

In this embodiment, the user (who may have some managerialresponsibility in the retail establishment, but may also be a retailestablishment analyst or even a shopper, or some other person interestedin analyzing the retail establishment or its products and processes)begins use of system in a “user session” with the mixed reality device13 a initiating the session by a switch on the device, a voice command,and/or a hand gesture. The mixed reality device 13 a includes a motionsensor that awakens the device 13 a (i.e., loads operating systemcomponents and prepares for input) when the device 13 a senses motion.The mixed reality device 13 a may require input of a user ID andpassword to enable further operation and interaction with the user. AnAR/VR session is initiated by the user via a menu user interface (notshown) of the mixed reality device 13 a.

As discussed the mixed reality device 13 a operate in various modes, andbased on the mode of operation (and the location of the user, and theorientation of the mixed reality device the AR/VR session manager 80chooses virtual items and context-relevant information to show to theuser on the screen.

One mode is a “loss prevention management” mode. In such a mode, theuser may approach a particular retail shelf and inspect a retail item,at which time the AR/VR session manager 80 sends information about theitem's inventory shrinkage (that is, difference between item sales anditem inventory replenishment integrated over a period of time). The usermay have several items in their field of view, and the device's userinterface may display menu options that allow the user to highlight aspecific item, or a group of items, and display information for avariable period of time which is also selected using the interface menuitems.

Inventory shrinkage analysis is accomplished when the AR/VR sessionmanager highlights items above some shrinkage threshold (measured insome meaningful metric such as absolute dollars lost, or percentincrease in shrinkage over some historical average or benchmark). Forexample, the user might walk down the aisle of the retail establishmentand as they do so, various retail items on the retail establishmentshelves come into their (physical) field of view. The Mixed realitydevice 13 a's camera sends these images back to the session manager 80via the web connection and the session manager identifies the items andcompares sales to current inventory levels (as determined by item-levelRFID scans or some other real-time or quasi-real-time inventorymeasurement technology). If the shrinkage or inventory loss level for aparticular item in view is in excess of a pre-determined threshold(selected by the user at the beginning of the session, or in somepre-configuration file used during initiation of the session), the itemis highlighted in red or outlined in some manner which emphasizes itslocation on the viewer of the mixed reality device 13 a. That is, theitem “glows” in some manner in the view of the user. The user may thenselect the item using voice or hand gestures and, as a result, see moredetailed information such as shrinkage data, shrinkage calculation inputdata, historical shrinkage data, comparisons of this data to data fromother retail establishments, retail categories, competing brands, and soforth.

Another mode is a comparative mode, where the user may view comparativeinformation for an item relative to other items in the retailestablishment in its inventory category, department, or relative to allitems in the retail establishment, or relative to this item or group ofitems in other retail establishments or groups of retail establishments(e.g., the retail establishment's regional division).

Another mode, as mentioned above allows the user to use the deviceinterface to enter notes as the user reviews the real and virtualobjects and information presented.

Another mode, as mentioned above allows the user to use the device whileengaged in such an AR/VR session to use standard voice communications orvoice-to-chat technology available on the mixed reality devices tocommunicate with a second (remote) user or group of users about whatthey are seeing and thinking, or to compose emails or text messages forimmediate or future sending.

The above mentioned application involves the session manager when inloss prevention management mode. Other modes might include“merchandising mode” or “planogram mode” in which the items in view ofthe user are studied with respect to their location on the shelf and inthe retail establishment, and how that relates to quality of sales.Optionally the session manager could operate in generic or “mixed” modein which any unusual information about an item is virtualized as somevisual object and presented to their view. Specifically, and usingmerchandising mode as an example, the user might consider an item inview, and highlight it with a hand gesture or voice command. Thedevice's user interface might then give graphics showing excursion aboveor below anticipated sales for a selected period of time, salescomparisons with other directly competing brands, other items in thesales category, the same item in other retail establishments in thevicinity, region, or nation, or comparisons with other sales benchmarks.A very valuable comparison is the comparison of sales (turnoverfrequency) of that item in its current shelf location, compared withsales in other past locations in the same retail establishment. As inthe case of the inventory shrinkage mode, the user may not need toselect specific items when doing a retail establishment analysis inmerchandising mode. That is, the user may simply walk up and down theretail establishment aisles and as exceptional items come into the fieldof view, the AR/VR session manager notes their exceptionality withrespect to one of the metrics and benchmarks mentioned above andhighlights the item in the field of view in the mixed reality deviceviewer.

Another application involves the analysis of the effectiveness of salespromotions. When in the merchandising mode or perhaps a specialpromotions analysis mode, the user can view an item and see projected bythe AR/VR session manager onto the device screen information related tosales (turnover frequency) relative to changes in retail price overselectable periods of time (in this retail establishment or a collectionof comparable retail establishments).

A particular application of the AR/VR system related to retailestablishment promotions is its correlation to foot traffic in eachaisle. As the user stands in a given location in an aisle, he or sheshould be able to see (based on menu selections via the device's userinterface) dwell time in front of each shelf location (of a selectedarea size, over a selected period of time) as a function of particularpromotions. For instance, the spot on the floor might show variousshades of green or red depending on whether foot traffic is a little ora lot above or below normal during the promotions period of the selectedretail item.

The AR/VR system may also generate a highlight on an area rendered inthe mixed reality device, where a retail establishment promotion issupposed to be in place at a given date and time, but is not in place.For example, a promotion may be scheduled to be on an aisle end-cap at9:00 am on Tuesday. If the promotion is not in place at the location andscheduled time, the AR/VR system can generate an alert that is renderedon the mixed reality device that could include a description and/orimage of the missing promotion and a proposed email or text to theperson in the retail establishment responsible. Using this function, andsimilar functionality related to planned advertising, price changes,warnings, and so forth, the user may simply walk through the retailestablishment and see (and record for later viewing, logging, orreporting) visual representations of differences between the actual andintended state of the retail establishment.

In another embodiment, the system of FIGS. 3 and 5 and the application(FIG. 4) described above are enhanced using an (artificial intelligencebased) cognitive assistant.

Referring now to FIG. 6, a cognitive agent 120 is shown. The cognitiveagent 120 is an example of one way of implementing such an agent. Thatis, those skilled in the art of expert systems, natural languageprocessing (NLP) based cognitive agents, and similar AI based agentswill be able to immediately envision various permutations of an agentwhich enables the applications described below.

Input text 121 (such as a question: “Has shrinkage for this itemincreased more than expected this month?”) enters the cognitive agentvia the NLP pre-processor 122. In voice based input embodiments, voicedata are converted to text by a voice to text sub-system 122 a usingstandard voice recognition technology. The NLP pre-processor 122 parsesthe sentence into “tagged” words and phrases (i.e., the NLPpre-processor 122 labels each word with its part of speech such as“noun”, “verb-transitive”, and so forth). The output of thepre-processor 122 is a first data structure that represents the inputtext with words and phrases connected to each other and annotated withindicators of meaning (or “sense” or “context”) pointing to the wordsand phrases. These meanings are extracted or determined using statisticsand pattern based comparisons to a semantic network 124, which alongwith the lexicon 126, retail establishments, includes the cognitiveagent's knowledge base of language (domain knowledge ontologies 128).

The representation of the input text (e.g., the first data structurewith determined meanings) is passed to a response macro-planner 130 thatdetermines information content of a response from the cognitive agent120. The macro-planner 130 can use non-monotonic planning with defaultlogic and patterns or templates (from episodic memory 132) and alsodomain knowledge 128 to produce a second data structure that containsthe response. This data structure does not contain “sentences” per se,but rather concepts and concept qualifiers (for time, quantity, type anddegree of emphasis, etc.) that together comprise the logic behind theresponse from the cognitive agent 120. This second data structure ispassed to a micro-planner 134 that produces sentences from the seconddata structure for the response from cognitive agent 120 using thelanguage knowledge contained in the lexicon 126 and semantic network124. The sentences can be “polished,” e.g., corrected, for grammar,(e.g., tense, singular/plural matching, etc.) in a finisher 136 thatoutputs text output 137. Domain knowledge ontologies and episodic memorymay be enhanced over time as the cognitive agent is given newinformation, and as old faulty information is corrected. That is, thesecan be written to selectively by “actuator” functionality in anextension of the response macro-planner.

The AR/VR session manager 80 may generate input text that is sentdirectly to the cognitive agent 120 in order to apprise the cognitiveagent of the context of the conversation with the user of the mixedreality device 13 a and resolve any ambiguities. For instance, when theuser asks the cognitive agent 120: “Has shrinkage for this itemincreased more than expected this month?” the cognitive agent 120 maynot know to what “this” refers, and will also not know the time periodin question without further information. The agent could ask the userfor time and item ID clarification, but it is more convenient for theuser if the cognitive agent 120 is passed information directly from thesession manager 80 about the item that is highlighted, and what timeperiod is selected or configured. The session manager 80 makes thesession state visible to the cognitive agent 120, via a web-service thatis not controlled by or visible to the user.

The knowledge base of the cognitive agent may be specifically trainedfor the particular retailer's markets, key customer base and associateddemographics, item types and uses, and any other knowledgecategorization which makes it easier for the agent to narrow questionsand better interpret the intention behind the user's questions. Thereare many questions that might be asked by the user of the mixed realitydevice 13 a of the cognitive agent 120.

Using the cognitive agent 120, the sales promotion application 70 can beenhanced as the cognitive agent 120 allows the user to ask questionsdirectly through the mixed reality device 13 a, which are sent as inputto the AR/VR session manager system 80 (see FIG. 7).

The user can considers an item and the accompanying information on salesvs. promotional price and can drill down to ask the cognitive agent 120(via the voice-to-chat interface) various questions such as:

(1) What was overall retail establishment performance like in thisretail establishment over the same period of time?

(2) What was the weather like?

(3) What were overall retail sales (in grocery, or apparel, or whatevercategory) like over the same period of time, relative to the previousyear?

(4) What was the foot traffic in this particular aisle of the retailestablishment, during the top six shopping hours of each Saturday, forthe 10 Saturdays previous to the beginning of the promotion, and foreach Saturday during the current promotion, and is there a statisticallymeaningful correlation between that traffic change and the promotion?

These are merely examples of questions that could be asked, as there aremany other questions that might be asked of the cognitive agent 120. Thecognitive agent 120 is made sufficiently flexible in its ability toprocess and interpret natural language, and it is sufficiently broad inits retail practices knowledge base that it can answer many and any suchquestions without requiring an enumeration and “hard coding” of specificquestions.

The cognitive agent 120 answers these questions by use of an ArtificialIntelligence platform that takes the output of the cognitive agent 120and parses words in the output. Such capability has been demonstrated toa limited extent by the Apple Siri platform, and to a much greaterextent by IBM's Watson AI engine and IPSoft's Amelia cognitive agent.The cognitive agent 120 can answer an actual question “where hasshrinkage increased over the prior month” by session manger system 80having a sub system “input analyzer 86 a” that processes input from thecognitive agent 120 and recognizes that shrinkage is being askedaccesses current and historical data, calculates the shrinkage if anyand delivers the calculated analytics back to the cognitive 120 to placethat value into a sentence that is rendered back to the user.

Another embodiment involves the visual correlation of item sales. Forexample, if the user of the mixed reality device 13 a highlights aparticular item on a shelf or display, the device 13 a may show allitems in the vicinity whose sales correlate most closely with theselected item. Another embodiment specifically includes an applicationof the described system in which the sales, shrinkage, or associatedfoot traffic correlations for a particular retail item or group of itemsis shown to the user, via the mixed reality device 13 a, for variousalternative retail establishment layouts including but not limited tothe current (physical) retail establishment layout. The user or usersengage in collaborate session to compare retail establishmentperformance impacts for those layouts, and investigate (particularlywhen the cognitive agent 120 is used) deep implications of retail dataanalytics applied to retail establishments with varying retailestablishment layouts that are shown visually on the mixed realitydevice 13 a.

Also are embodiments in which a previously recorded session for aparticular retail establishment is replayed to simulate the retailestablishment visit for the original user or a new user. One suchembodiment includes the case where images are captured (for example,while the user walks through the retail establishment) anddata/information virtualization and projection is applied later toproduce the augmented walk-through session. Another embodiment includesa session (produced by any of the techniques described above) involvinga series of real and virtual objects presented in the session, withsubsequent application of the cognitive agent 120 to answer userquestions and record user thoughts and expertise as the user watches thepreviously recorded session. That is, the cognitive agent 120 generatesa data structure that represents the answer to the question in the voiceinput. The cognitive agent 120 determines the meaning of the questionand forms queries to execute merchandizing algorithms that answer thequestion, and packages the answer into text that is rendered to theuser.

Servers can be any of a variety of computing devices capable ofreceiving information, such as a server, a distributed computing system10, a rack-mounted server and so forth. Server may be a single server ora group of servers that are at a same location or at differentlocations. Servers can receive information from client device userdevice via interfaces. Interfaces can be any type of interface capableof receiving information over a network, such as an Ethernet interface,a wireless networking interface, a fiber-optic networking interface, amodem, and so forth. Server also includes a processor and memory and abus system including, for example, an information bus and a motherboard,can be used to establish and to control information communicationbetween the components of server.

Processor may include one or more microprocessors. Generally, processormay include any appropriate processor and/or logic that is capable ofreceiving and storing information, and of communicating over a network(not shown). Memory can include a hard drive and a random access memorystorage device, such as a dynamic random access memory computer readablehardware storage devices and media and other types of non-transitorystorage devices.

Embodiments can be implemented in digital electronic circuitry, or incomputer hardware, firmware, software, or in combinations thereof.Computer programs can be implemented in a high-level procedural orobject oriented programming language, or in assembly or machine languageif desired; and in any case, the language can be a compiled orinterpreted language. Suitable processors include, by way of example,both general and special purpose microprocessors. Generally, a processorwill receive instructions and information from a read-only memory and/ora random access memory. Generally, a computer will include one or moremass storage devices for storing information files; such devices includemagnetic disks, such as internal hard disks and removable disks;magneto-optical disks; and optical disks. Storage devices suitable fortangibly embodying computer program instructions and information includeall forms of non-volatile memory, including by way of examplesemiconductor memory devices, such as EPROM, EEPROM, and flash memorydevices; magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD_ROM disks. Any of the foregoing can besupplemented by, or incorporated in, ASICs (application-specificintegrated circuits).

Other embodiments are within the scope and spirit of the descriptionclaims. For example, due to the nature of software, functions describedabove can be implemented using software, hardware, firmware, hardwiring,or combinations of any of these. Features implementing functions mayalso be physically located at various positions, including beingdistributed such that portions of functions are implemented at differentphysical locations. For example remote users may use instead of a mixedreality device and session manager as described above, a computerapplication with a graphical user interface that projects some versionor portrayal of the in-retail establishment user's screen and displayedinformation. Other embodiments therefore are within the scope of thefollowing claims.

What is claimed is:
 1. A system comprises: a server system comprising:one or more processor devices; memory in communication with the one ormore processor devices; and a storage device that stores a program ofcomputing instructions for execution by the processor using the memory,the program comprising instructions configured to cause the processorto: receive images from a mixed reality device, which images includingrepresentations of physical objects; receive voice data from the mixedreality device; execute a cognitive agent to process the voice datareceived from the mixed reality device; determine location informationto determine a correspondence between the representations of thephysical objects in the images and stored information concerning the setof physical objects; generate a set of virtual objects that include datadescribing merchandising process concepts regarding the set of physicalobjects, based on input queries received in the voice data; and send theset of virtual objects to the mixed reality device.
 2. The system ofclaim 1 comprises: a voice to text translator.
 3. The system of claim 2wherein the server system executes the cognitive agent that includes: anatural language process pre-processor that parses text input into“tagged” words and phrases to provide a first data structure thatrepresents the input text with words and phrases connected to each otherand annotated with indicators of meaning; and a response macro-plannerthat determines information content of a response from therepresentation of the input text to produce a second data structure thatcontains the response in a form that contains concepts and conceptqualifiers that comprise logic behind the response; and a micro-plannerthat produces sentences from the second data structure for the responsefrom cognitive agent.
 4. The system of claim 3 wherein the computingsystem configured to execute the cognitive agent is further configuredto: receive user inputs regarding the physical items.
 5. The system ofclaim 4 wherein the received user inputs regarding the physical itemsare highlights applied to a selected one or more of the physical items.6. The system of claim 1 wherein the computing system, configured toexecute the cognitive agent, is further configured to: analyze the userinputs; and resolve ambiguities in the user inputs by accessing datadirectly from the mixed reality device, which data specifies thephysical objects by having a highlight applied on the specified one ofthe physical objects.
 7. The system of claim 1 wherein cognitive agentis configured to: receive voice input in the form of a questionregarding a product displayed in the mixed reality device; generate astructure that represents an answer to the question in the voice input,by executing the cognitive agent to determine meaning of the questionand forming queries to execute merchandizing algorithms that answer thequestion.
 8. The system of claim 1 wherein the server system is furtherconfigured to: send the answer to the mixed reality device.
 9. Thesystem of claim 1 wherein the server system is further configured to:execute one or more merchandizing algorithms that apply the data derivedfrom the question and that apply data corresponding to the one or moremerchandizing parameters associated with the set of physical objects toprovide as the output data one or more merchandizing values associatedwith one or more of the physical objects in the view of the set ofphysical objects.
 10. A computer implemented method, the methodcomprising: receiving by a server system, images from a mixed realitydevice, which images including representations of physical objects;receiving by a server system, voice data from the mixed reality device;executing by a server system, a cognitive agent to process the voicedata received from the mixed reality device; determining by a serversystem, location information to determine a correspondence between therepresentations of the physical objects in the images and storedinformation concerning the set of physical objects; generating by aserver system, a set of virtual objects that include data describingmerchandising process concepts regarding the set of physical objects,based on input queries received in the voice data; and sending by aserver system, the set of virtual objects to the mixed reality device.11. The method of claim 10, further comprising: executing by the serversystem a cognitive agent.
 12. The method of claim 10, furthercomprising: analyzing by the server system the user inputs; andresolving by the server system ambiguities in the user inputs byaccessing data directly from the mixed reality device, which dataspecifies the physical objects by having a highlight applied on thespecified one of the physical objects.
 13. The method of claim 10,further comprising: executing by the server system one or moremerchandizing algorithms that apply the data derived from the questionand that apply data corresponding to the one or more merchandizingparameters associated with the set of physical objects to provide as theoutput data one or more merchandizing values associated with one or moreof the physical objects in the view of the set of physical objects.